Snakemake Report

Loading Snakemake Report...

Please enable Javascript in your browser to see this report.

Loading 4.6 MB. For large reports, this can take a while.

Workflow

Click the nodes to obtain details about each step.

pandas_profiling

Statistics

If the workflow has been executed in cluster/cloud, runtimes include the waiting time in the queue.

Configuration

File Code

config.yaml

# config used for Snakemake 


remote: true

gcloud:
    bucket: "lvt-schiphol-assignment-snakemake"
    credentials: "keys/bucket-access.json"


mlflow:
    experiment: "schiphol-snakemake"
    tracking_uri: "lvt-schiphol-assignment-snakemake/mlruns"

raw_files:
    - "flights.csv"
    - "airports.csv"


trelliscope:
    path: "trelliscope"

train_test:
    test_size:
        - 0.2
    strategy:
        - sample
        - timeseries


models:
  - baseline_average
  # - catboost_simple

Jobs 2

Input files

{bucket}/data/raw/{table_name}.csv
scripts/explore__pandas_profiling.ipynb

Output files

{bucket}/reports/profiling_reports/{table_name}.html

Code

1	`papermill {input.notebook} "{log.notebook}" -k {params.env} --inject-paths -p input_file "{input.data}" -p output_file "{output.html_report}" -p profiling_config "scripts/profiling_config.yml"`

Jobs 2

Input files

{bucket}/data/model_input/delays_extended_input.csv
{bucket}/data/model_input/train_test__{test_size}__{strategy}.csv
scripts/model__baseline_average.ipynb

Output files

{bucket}/data/model_output/baseline_average__{test_size}__{strategy}/predictions.csv

Code

papermill {input.notebook} "{log.notebook}" -k {params.env} --inject-paths -p input_file "{input.data}" -p train_test_file "{input.train_test}" -p output_predictions "{output.predictions}" -p mlflow_tracking_uri "{params.mlflow_uri}" -p mlflow_experiment "{params.mlflow_experiment}" -p mlflow_run "{params.mlflow_run}"

Jobs 1

Input files

{bucket}/data/model_input/delays_base_input.csv
scripts/feature__rolling_mean_delay.ipynb

Output files

{bucket}/data/model_input/features/rolling_mean_delay__{freq}__{window}.csv

Code

1	`papermill {input.notebook} "{log.notebook}" -k {params.env} --inject-paths -p input_file "{input.data}" -p output_file "{output.data}" -p freq "{wildcards.freq}" -p window "{wildcards.window}"`

Jobs 1

Input files

{bucket}/data/model_input/delays_base_input.csv
{bucket}/data/model_input/features/schedule_time_features.csv
{bucket}/data/model_input/features/route_destinations.csv
scripts/preprocess__extend_base_with_features.ipynb

Output files

{bucket}/data/model_input/delays_extended_input.csv

Code

1	`papermill {input.notebook} "{log.notebook}" -k {params.env} --inject-paths -p base_file "{input.base}" -p features "{input.time_features}+{input.route_features}" -p output_file "{output.data}"`

Jobs 2

Input files

{bucket}/data/model_input/delays_base_input.csv
scripts/preprocess__train_test_split.ipynb

Output files

{bucket}/data/model_input/train_test__{test_size}__{strategy}.csv

Code

papermill {input.notebook} "{log.notebook}" -k {params.env} --inject-paths -p input_file "{input.data}" -p output_file "{output.data}" -p test_size {wildcards.test_size} -p strategy {wildcards.strategy}

Jobs 1

Input files

{bucket}/data/raw/flights.csv
scripts/preprocess__base_model_input.ipynb

Output files

{bucket}/data/model_input/delays_base_input.csv

Code

1	`papermill {input.notebook} "{log.notebook}" -k {params.env} --inject-paths -p input_file "{input.data}" -p output_file "{output.data}"`

Jobs 1

Input files

{bucket}/data/raw/flights.csv
{bucket}/data/raw/airports.csv
scripts/feature__route_destinations.ipynb

Output files

{bucket}/data/model_input/features/route_destinations.csv

Code

1	`papermill {input.notebook} "{log.notebook}" -k {params.env} --inject-paths -p flights_file "{input.flights}" -p airports_file "{input.airports}" -p output_file "{output.data}"`